A brain-inspired algorithm for training highly sparse neural networks

Conjugate gradient algorithm for training neural networks

Training of Feed-Forward Neural Networks for Pattern-Classification Applications Using Music Inspired Algorithm

There have been numerous biologically inspired algorithms used to train feed-forward artificial neural networks such as generic algorithms, particle swarm optimization and ant colony optimization. The Harmony Search (HS) algorithm is a stochastic meta-heuristic that is inspired from the improvisation process of musicians. HS is used as an optimization method and reported to be a competitive alt...

DSD: Dense-Sparse-Dense Training for Deep Neural Networks

Modern deep neural networks have a large number of parameters, making them very hard to train. We propose DSD, a dense-sparse-dense training flow, for regularizing deep neural networks and achieving better optimization performance. In the first D (Dense) step, we train a dense network to learn connection weights and importance. In the S (Sparse) step, we regularize the network by pruning the un...

Provable Methods for Training Neural Networks with Sparse Connectivity

We provide novel guaranteed approaches for training feedforward neural networks with sparse connectivity. We leverage on the techniques developed previously for learning linear networks and show that they can also be effectively adopted to learn non-linear networks. We operate on the moments involving label and the score function of the input, and show that their factorization provably yields t...

A new training algorithm for feedforward neural networks